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PREFACE 


ThiF  Memorandum  describes  progress  in  the  design  and 
construction  of  hybrid  computing  hardware  for  experimenta¬ 
tion  with  neural  nets. 

The  primary  aim  of  the  overall  project-conducted  for 
the  Advanced  Research  Projects  Agency— is  to  mimic  observed 
gross  behavioral  activity  of  neurophysiological  systems, 

with  the  hope  of  exhibiting  elementary  motivated  behavioral 
patterns. 
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SUMMARY 


This  Memorandum  presents  hardware  designed  to  provide 
basic  building  blocks  from  which  large  general  neural  net- 
works  can  be  constructed. 

An  introduction  discusses  the  relationship  of  such 
constructions  to  automata  theory  and  their  possible  use¬ 
fulness  m  the  building  of  neural  nets.  In  the  main  body 
of  the  text,  a  simple  network  is  constructed,  containing 
by  way  of  example  at  least  one  representative  of  each  of 
five  basic  types  of  circuits.  Also  included  is  informa¬ 
tion  on  the  logical  properties  of  the  building  blocks. 

An  Appendix  gives  detailed  descriptions  of  the  required 
circuitry  and  control  functions. 
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HARDWARE  AIDS  FOR  AUTOMATA  DESIGN 
INTRODUCTION 

Among  the  fundamental  efforts  toward  a  logical  de¬ 
scription  of  neuron-like  elements  are  the  works  of 
Kleene  [1]  and  von  Neumann  [2],  both  following  the  basic 
paper  of  McCulloch  and  Pitts  [3], 

Kleene  reviews  the  Mr.Culloch-Pitts  concept  of  a  nerve 
net  constructed  of  elementary  neurons.  In  this  scheme,  a 
neuron  consists  of  a  body,  or,  soma,  from,  which  axons  lead 
to  one  or  more  end  bulbs.  A  nerve  net  is  an  arrangement 
of  both  "input"  and  "inner"  neurons,  where  any  given  axon 
impinges  on  not  more  than  one  other  cell  body,  and  where 
each  synapse  is  either  excitatory  or  inhibitory.  With 
these  building  blocks,  one  can  perform  such  operations  as 
logical  conjunction  and  disjunction,  and  also  work  with 
delays.  Kleene  thoroughly  discusses  the  representability 
of  events  and  concepts  of  finite  automata. 

Von  Neumann  discusses  problems  of  automata  synthesis 
and  emphasizes  the  role  of  error  in  the  physical  implementa¬ 
tion  of  probabilistic  logics. 

Harmon  and  Lewis  [4]  have  recently  given  a  definitive 
of  neural  modeling  intended  to  more  exactly  repre¬ 
sent  physiological  phenomena. 

While  such  discussions  are  important  and  necessary, 
a  knowledge  of  the  logical  properties  of  individual  ele¬ 
ments,  or  small  groups  of  elements,  is  often  of  limited 
value  in  the  synthesis  of  large-scale  automata  designed 

: k 

By  "large-scale  automata"  is  meant  a  network  com¬ 
prised  of  a  large  number  of  elements  where  each  element 
performs  one  of  a  small  number  of  functions. 
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to  show  explicitly  input/output  behavior.  That  is,  a 
description  of  the  psychology,  or  gross  behavior,  of  an 
automaton  is  often  only  distantly  related  to  a  description 
of  its  neurophysiology.  Therefore,  when  constructing 
large-scale  automata,  one  must  usually  first  synthesize  by 
constructing  trial  interconnections  between  his  chosen 
building  blocks,  then  analyze  the  behavior  of  his  construc¬ 
tion-in  essence,  a  "synthesis  by  analysis."  The  initial 
selection  of  connections  rarely  models  the  desired  input/ 
output  behavior.  This  process,  however,  may  reveal  ways 
to  improve  the  circuitry  in  the  next  iteration. 

Evidently,  formal  graphical  or  tabular  algorithms  are 
needed  to  describe  routines  for  the  synthesis  of  automata 
on  a  behavioristic  level.  Such  algorithms  may  be  long 
coming ,  if  indeed  at  all,  despite  a  designer's  intuition- 
when  constructing  automata  with  pencil  and  paper-- that  he 
is  calling  upon  a  describable,  finite  set  of  conceptual 

operations . 

This  Memorandum  takes  a  less  sophisticated  but  re¬ 
alizable  approach- -concentrating  on  the  acceleration  of 
the  analysis  portion  of  the  "synthesis  by  analysis"  routine 
This  is,  admittedly,  a  concession;  however,  any  scheme 
providing  a  more  immediate  reaction  to  the  manipulations 
of  the  designer  should  materially  minimize  interruptions 
during  pursuit  of  design  goals.  No  such  scheme  or  device 
is  currently  in  use  (which  may  partially  account  for  the 
lack  of  understanding  of  the  problem-solving  capabilities 
of  automata,  or  more  generally,  about  information  flow 
in  parallel  systems). 
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The  basic  motivation  for  the  circuits  described  here 
is  provided  by  Paxson,  who  offers  the  following  comments: 


...  a  new  experimental  tool  is  needed.  Simulation 
of  neural  nets  on  a  digital  computer  is  inefficient. 
The  serial  computation  of  a  structure  of  parallel 
operations  is  expensive  in  preparation  and  execution 
times.  Nor  is  any  use  made  of  the  arithmetic  pre¬ 
cision  inherent  in  the  digital  computer- -it.  is 
actually  inappropriate.  Finally,  the  turnaround 
time  to  change  parameters  and  analyze  the  conse¬ 
quences  is  excessive--four  to  twenty-four  hours, 
and  much  expensively  produced  paper  is  wasted. 
Coupling  between  experimenter  and  computer  is  not 
close  enough. 

Special  purpose  analogue  equipment  is  needed.  A 
particular  net  should  be  set  up  and  modified  on  this 
equipment  under  the  experimenter's  own  hands.  Com¬ 
ponents  should  reproduce  the  behavior  described  in 
the  section  above  on  neural  nets,  and  would  not 
resemble  closely  past  neuromimes  (TAYLOR,  HARMON) 
that  were  designed  to  reproduce  spike  trains  and 
simulate  other  features  of  real  neurons.  We  hope 
to  investigate  the  design  of  such  equipment.* 


A  discussion  of  the  circuit  design  will  be  presented 
in  three  parts: 

1)  General  design  principles; 

2)  Logical  properties  of  the  building  blocks; 

3)  A  detailed  description  of  the  actual  circuitry 
(Appendix) . 


Ref.  5 ,  p . 


47. 
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GENERAL  DE  IGN  PRINCIPLES 

The  system  contains  a  mixture  of  analog  and  digital 
characteristics.  Threshold  logic  circuits  have  analog 
inputs  and  digital  outputs.  A  second  class  of  circuits, 
having  digital  inputs  and  analog  outputs ,  defines  how 
signals  are  altered  in  transit  from  one  threshold  circuit 
to  another.  The  analog  inf on  ition  in  circulation,  altered 
at  discrete  time  intervals,  produces  staircase  waveforms , 

A  problem  is  run  over  a  number  of  time  intervals  set  info 
the  machine  by  the  operator.  The  machine  operates  in  a 
repetitive  mode  similar  to  current  analog  computing,  which 
continually  displays  the  solution.  Thus,  if  the  interval 
counter  is  set  to  run  a  time  units,  the  problem  will  run 
through  n  time  units,  the  entire  system  is  then  reset,  and 
thr  problem  is  run  again  through  n  time  units,  etc.  This 
operation  continues  until  interrupted  by  the  operator,  who 
stops  the  system  or  resets  n  to  a  new  value. 

The  system's  basic  timing  Is  delivered  to  the  control 
circuitry  by  a  pulse  generator  or  clock  set  to  provide  a 
1  kc  square  wave  signal.  The  control  circuitry  monitors 
the  presentation  of  clock  pulses  to  the  network  and  re¬ 
sets  the  problem  run  at  the  appropriate  intervals. 

Tapping  into  a  shift  register  at  the  desired  interval 
achieves  delay  of  the  signal  pulse  propagated  in  a  particular 
circuit  (the  transit  time  from  point  to  point).  Since  the 
shift  register  derives  its  commands  from  the  clock,  the 
delay  of  a  given  signal  is  always  an  integral  multiple  of 
clock  periods  or  system  time  units  (see  Appendix). 
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The  circuits  themselves  represent  an  abstraction  of 
gross  neurophysiological  observations.  For  example, 
rectangular  pulses  and  ramp  pulses,  whose  durations  are 
proportional  to  firing  periods,  replace  the  normal 
"sequence  of  spikes"  representation  in  neural  nets. 

The  most  convenient  output  appears  to  be  a  multi¬ 
trace  scope  where  one  can  display  the  entire  history  of 
the  problem  at  selected  points  in  the  net. 

Components  and  circuitry  are  mounted  on  module  cards 
and  housed  in  a  19  in.  cage  (see  Frontispiece).  The 
intermodule  wiring  is  done  on  connector  pins  at  the  rear 
of  t:he  cage.  Next-generation  construction  may  use 
physically  smaller  building  blocks  allowing  units  to  be 
patched  on  a  large  plugboard,  thus  preserving  network 
topology.  Controls  for  parameter  settings  are  mounted 
on  the  ends  of  the  cards;  the  interval  counter  switch 
and  circuit  test  poincs  are  at  the  top  of  the  cage. 

LOGICAL  PROPERTIES  OF  BUILDING  BLOCKS 

The  design  involves  the  interconnection  of  two  basic 
kinds  of  functional  circuits: 

1)  Threshold  Circuits 

a)  Type  I — In  this  type  of  threshold  circuit  (Fig.  1) , 
the  digital  output  is  "1"  when  the  sum  of  all 
analog  inputs  exceeds  an  arbitrary  threshold  and 
"0"  at  all  other  times. 

b)  Type  II--This  type  of  threshold  device  fires  at 
an  arbitrary  setting,  but  sustains  firing  until 
the  sum  of  inputs  drops  below  a  second  arbitrary 
lower  threshold. 


Second  threshold 


Threshold  circuit 
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2)  Transfer  Circuits 


These  circuits  determine  how  waveforms  are  altered 
while  in  transit  from  one  threshold  circuit  to  the  next. 
There  are  four  kinds  of  transfer  circuits: 

a)  R  Circuit  (Reinforcer)--Generates  a  rectangular 
pulse  with  adjustable  amplitude  and  delay  when 
the  digital  input  is  a  "1"  (Fig.  2). 

b)  F  Circuit  (Fac  ,.litation)--Generates  a  sloping 
voltage  representing  facilitation  with  an  ad¬ 
justable  initial  value,  final  maximum  value, 
and  delay  when  the  input  is  "l15  (Fig.  3). 

c)  L  Circuit  (Learner)--Learning  occurs  only  during 
periods  when  both  the  learning  synapse  and  a 
reinforcing  synapse  are  active  (Fig.  4).  Between 
periods  of  reinforcement,  the  voltage  contributed 
by  the  learner  (or  which  would  be  contributed  if 
its  parent  cell  were  firing)  decays  exponentially 
to  become  an  asymptote,  once  the  latter  has  been 
exceeded.  Learned  synapses  do  not  reset.  Memory 
of  the  learning  synapse  status  is  maintained  in 
the  computation  so  that,  upon  new  reinforcement, 
the  learner  voltage  starts  at  its  current  value, 
provided  its  parent  cell  is  firing. 

d)  P  Circuit  (Probability) --Generates  an  output 
which  has  an  adjustable  amplitude  and  an  adjust¬ 
able  probability  (P)  of  occurring  when  the  input 
is  "1"  (Fig.  5).  The  probability,  P,  of  an  out¬ 
put,  A,  occurring  is  determined  by  sampling  a 
random  process  at  discrete  time  intervals  governed 
by  the  clock  (see  Appendix). 

All  of  the  above  circuits  have  been  constructed 
from  Fairchild  integrated  circuit  components ,  and  connected 
so  as  '  o  mimic  the  behavior  of  the  simple  demonstration 
neural  net  outlined  by  Paxson  [5]  (Fig.  6).  Circuits  A, 

B,  and  C  are  threshold  circuits,  arbitrarily  set  at  10 
units . 


-li¬ 


lt 

<  X 
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sign  study 
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The  notation  3,2/10  L  in  Fig.  6  represents  a  learner 
circuit  with  the  delay  or  transit  time  r  =  3,  ir ^tial 
voltage  =  2  and  final  voltage  =  10.  The  notation 
2P  indicates  that  an  amplitude  of  two  units  will  occur 
with  probability  P.  The  notation  5R  stands  for  a  rein¬ 
forcer  with  unit  delay  (t  =  1)  and  an  amplitude  of  5. 

The  notation  -5D  represents  a  defacilitation  (negative 
facilitation)  circuit  with  V.  =0,  t  =  1  and  a  V  *  -5 
units . 

The  waveforms  shown  in  Fig.  7  indicate  the  voltage¬ 
time  relationships  exhibited  by  the  various  circuit?  for 
the  one  prescribed  set  of  values  for  threshold,  delay, 
amplitude,  initial  and  final  value,  and  decay  settings. 
(Details  of  schematic  operation  are  discussed  in  the 
Appendix. ) 

On-line  graphical  facilities  may  also  prove  a  useful 
approach  to  "synthesis  by  analysis"  operations.  For 
example,  naming  and  recall  of  substructures,  useful  for 
recalling  operations  previously  carried  out,  may  be  more 
easily  realized  with  digital  machinery  (assuming  avail¬ 
ability  of  sufficient  space).  However,  the  principle 
advantage  of  the  present  hardware  approach  lies  in  its 
high  speed  and  consequent  uninterrupted  interaction  be¬ 
tween  operator  and  design  tool. 

Further,  this  design  has  enabled  us  to  specify  re¬ 
quired  circuitry  in  terms  of  current  technology,  and  should 
serve  as  a  guide  in  future  discussions  of  specific  forms 
of  automata  design  aids. 


Set  to  constant  12  units 


aveform  example 
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Appendix 

DETAILED  DESCRIPTION  OF  CIRCUITRY 


Five  basic  types  of  Fairchild  micrologic  integrated 
circuits  are  employed: 

1)  The  710  comparator  circuit  has  an  output  of  "1M 
(+3  volts)  when  me  input,  Pin  2,  is  greater  in 
magnitude  than  the  analog  input,  Pin  3. 


2)  The  914  dual  nand  gate  package  consists  of  two 
2-input  nand  gates  whose  connections  are  shown. 
For  example,  the  output  at  Pin  7  is  contingent 
upon  the  input  values  at  Pins  1  and  2,  as  shown 
in  the  following  table: 


Pin 

1 

Pin 

2 

Pin 

7 

0 

0 

1 

0 

1 

1 

1 

0 

I 

1 

1 

0 
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3)  The  923  J-K  flip-flop  has  set  and  clear  inputs 
at  Pins  1  and  3,  respectively,  and  set  and  reset 
outputs  at  Pins  6  and  5,  respectively.  For 
example,  the  output  at  Pin  7  (at  time  n+1)  is 
contingent  upon  the  inputs  at  Pins  1  and  3  (at 
time  n) ,  ii.  accordance  with  the  following  table- 
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4)  The  Fairchild  702  is  a  high-gain,  wideband 

amplifier  used  as  an  operational  amplifier  with 
inputs  on  Pins  2  and  3  and  output  at  Pin  7. 


5)  The  900  is  a  buffer  element  used  when  driving 
a  large  number  of  circuits  (i.e.,  when  large 
current  requirements  exist).  Input  and  output 
connections  are  at  Pins  j  and  5,  respectively. 
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THRESHOLD  CIRCUIT 

The  threshold  circuit  (Fig.  8)  is  forced  with  a  71C 
comparator  and  914  gates.  Input  voltages  are  summed  across 
the  input  resistors  and  presented  as  one  input  to  the  com¬ 
parator.  The  second  comparator  input  (Pin  2)  is  an  adjust¬ 
able  voltage  (see  IK  potentiometer)  which  controls  the 
threshold  of  the  device.  The  10K  feedback  resistor  from 
Pins  7  to  2  of  the  710  unit  minimizes  hysteresis.  The  914 
gates  provide  both  output  drive  and  a  method  for  gating 
off  the  output  during  reset  time  after  the  running  of  each 
problem  cycle. 

RFINFORCER  CIRCUIT 

The  basic  delay  function  in  the  reinforcer  circuit 
(Fig.  9)  is  achieved  by  tapping  off  the  desired  delays 
at  different  points  of  a  shift  register.  Delays  of  up 
to  four  time  units  are  provided  by  four  923  flip-flops. 

The  required  "true"  and  complementary  data  inputs  for  the 
four-stage  shift  register  are  provided  by  a  914  gate.  The 
system  clock  advances  the  register.  The  resistor- transistor 
network  (shown  at  the  bottom  of  Fig.  9)  then  adjusts 
amplitude  of  the  delayed  function. 

FACILITATION  CIRCUIT 

The  delay  in  the  facilitation  circuit  is  achieved  in 
the  same  way  as  in  the  reinforcer  circuit.  The  shift 
register  is  illustrated  at  the  bottom  of  the  schematic  in 
Fig.  10.  The  702  is  connected  as  a  summing  integrator, 


1 0K 


Reinforcer 


2N3638 


Facilitation  circuit 
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discharging  when  the  2N3638  transistor  shunting  che  feed¬ 
back  capacitor  is  shorted.  The  914  gating  circuits  compute 
the  time  of  discharge  by  logically  comparing  the  selected 
delay  time  with  the  next  successive  stage  of  the  shift 
register.  The  914  gating  circuits  similarly  compute  the 
time  at  which  the  desired  (V  initial)  is  applied  by 
considering  the  onset  of  the  selected  delay  and  the  onset 
of  delay  at  the  next  successive  stage.  The  10K  pot  (upper 
left  area  of  the  schematic)  adjusts  the  amplitude  of  V. . 

The  selected  amplitude  is  then  gated  into  one  of  the  three 
summing  resistors  of  the  702  at  the  appropriate  time.  The 
second  summing  resistor  is  used  as  a  balancing  input.  The 
lower  summing  resistor  feeds  clock  pulses  to  the  702  which, 
when  integrated,  result  in  the  basic  staircase  waveform 
at  the  702  output.  The  diode  circuitry  at  Pin  6  of  the 
comparator  provides  a  clamp  for  the  output  to  the  voltage 
(V  final)  set  by  the  10K  potentiometer  (right  side 
of  the  schematic). 

LEARNER  CIRCUIT 

The  computation  of  delay,  discharge  time,  and  time  at 
which  to  apply  a  voltage  are  all  accomplished  as  in  the 
facilitation  circuit.  However,  the  summing  inputs  to  the 
operational  amplifier  differ.  The  first  two  control  the 
sign  of  the  slope  of  the  staircase  waveforms.  That  is, 
starting  the  reinforcer  associated  with  the  learner  activates 
the  input,  R,  resulting  in  the  gating  of  clock  pulses  to  the 
first  summing  input  of  the  702.  If  the  R  line  is  not  acti¬ 
vated,  clock  pulses  of  reversed  polarity  are  presented  to  the 
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702  through  the  second  summing  input.  Thus,  the  output 
has  either  a  positive  or  negative  staircase  waveform, 
according  to  the  presence  or  absence  of  the  reinforcer, 
respectively . 

The  third  and  fourth  summing  inputs  are  used  for 
balancing  and  for  the  presentation  of  V^,  respectively. 

The  output  from  the  702  operational  amplifier  feeds  Pin  2 
of  a  710  comparator.  The  voltage  on  Pin  3  of  this  com¬ 
parator  is  controlled  by  a  voltage  value  fed  through  an 
emitter  follower  from  a  2.7K  potentiometer.  When  the 
output  of  the  702  exceeds  the  voltage  setting  on  Pin  3 
of  the  710,  the  710  fires.  Then  the  transistor  circuitry 
fed  by  the  710  behaves  as  a  latching  circuit,  permanently 
clamping  the  702  output  to  the  voltage  setting  on  the 
2.7K  potentiometer.  Thus,  once  the  702  output  exceeds  the 
desired  learning  plateau,  it  remains  clamped  at  the  value 
so  that  the  output  may  climb  higher,  but  may  never  drop 
below,  the  acquired  plateau  (Fig.  11). 

PROBABILITY  CIRCUIT 

A  random  function  for  the  probability  circuit  (Fig. 

12)  is  obtained  by  connecting  two  702  high-gain  amplifiers 
in  series.  The  noise  generated  by  the  first  transistor 
in  the  first  702  is  amplified  and  fed  to  one  input  of  a 
710  comparator.  The  second  input  to  the  comparator,  Pin  3, 
comes  from  a  100K  potentiometer  which  sets  the  probability 
value  by  determining  at  what  amplitude  of  the  random  func¬ 
tion  the  710  will  be  activated.  The  clock  then  samples 
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the  condition  of  the  710  output  at  discrete  intervals. 
Finally,  a  small  three-transistor  network  (right  side  of 
Fig.  12)  controls  the  amplitude  of  the  output. 

CONTROL  CIRCUITRY 

A  two-stage,  10-position,  manually  operated  switch 
is  wired  to  provide  binary-coded  decimal  (BCD)  outputs 
(Fig,  13).  These  outputs  are  compared,  through  negated 
“exclusive  or"  logic  on  a  bit-by-bit  basis,  with  the 
current  number  in  a  running  BCD  counter.  If  the  two 
numbers  are  equal,  an  output  pulse  appears  indicating 
that  the  BCD  count  is  equal  to  the  number  set  in  the 
switch.  When  a  match  is  found,  the  BCD  counter  and  each 
circuit  in  the  system  is  reset,  allowing  the  problem  to 
be  rerun. 

A  stop-go  switch  sets  a  923  flip-flop  whose  output 
controls  the  transmission  of  the  clock  into  the  control 
circuits.  A  900  booster  provides  drive  for  the  reset 
line . 
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Fig.  13  —  Control  circuit 
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